A Contextual Discretization Framework for Compressing Recurrent Neural Networks

In this paper, we address the issue of training Recurrent Neural Networks with binary weights and introduce a novel Contextualized Discretization (CD) framework and showcase its effectiveness across multiple RNN architectures and two disparate tasks. We also propose a modified GRU architecture that allows harnessing the CD method and reclaim the exclusive usage of weights in {−1, 1}, which in turn reduces the number of power-two bit multiplications from O(n) to O(n).